Machine Learning Based Profile Driven Scheduling Algorithm

Clusters of commodity servers are increasingly the platform of choice for running computationally and IO intensive jobs in a variety of industries. It is expected that using clusters will reduce the average job response time. But improper submission of jobs to clusters may lead to two problems, first it leads to blocking of jobs (waiting for results from other jobs) second it leads to disturbing other jobs (i.e. other jobs may be blocked due to submission). Effective utilization of the resources in clusters can help to balance the load and avoid situations like slow run of systems. This paper addresses the principle of effective utilization of cluster resources by Machine Learning based profile driven scheduling. It avoids the above problems by allocating jobs to resources of cluster based on the SVM prediction profiling results. Some of the machines in the cluster run the IO bound jobs effectively with minimum waiting time and with the minimum execution time, while in some other machines run the CPU bound jobs effectively with minimum waiting time and with the minimum execution time. The system statistics like CPU utilization time, Memory space utilized, User and System time utilization are used as the parameters for Profiling. Job dependency analysis is used to prevent dependent jobs to keep blocking and disturbing other jobs. This paper uses the Resource profiling results based on machine learning prediction and Offline Job profiling to perform job allocation onto the resources of cluster.